Communications method involving groups of processors of a distributed computing environment

ABSTRACT

In a distributed computing environment having a plurality of groups of processors, each processor group maintains its own set of ordered messages. A message is sent to a group of processors. In particular, a request to multicast a message is sent to a leader of the group of processors, i.e., a group leader, and the group leader assigns a sequence number to the message to be sent, thereby providing an ordered message. Then, the group leader multicasts the ordered message to the group of processors. When the processors of the group receive the ordered message, each determines whether the message is in proper sequence order. If the message is out of order for a particular processor indicating that a message has been missed, then that particular processor requests the missing message from a processor within the group.

TECHNICAL FIELD

This invention relates, in general, to distributed computingenvironments and, in particular, to each group of processors of thedistributed computing environment maintaining its own ordered set ofmessages.

CROSS REFERENCE TO RELATED APPLICATIONS

This application contains subject matter which is related to the subjectmatter of the following applications, which are assigned to the sameassignee of this application and are filed on the same day as thisapplication. Each of the below listed applications is herebyincorporated herein by reference:

"An Application Programming Interface Unifying Multiple Mechanisms," byP. R. Badovinatz et al., Ser. No. 08/960,074 which is a file wrappercontinuation of Ser. No. 08/640,305, now abandoned;

"A Method For Managing Membership Of A Group Of Processors In ADistributed Computing Environment," by P. R. Badovinatz et al., Ser. No.08/640,412;

"Utilizing Batch Requests To Present Membership Changes To ProcessGroups," by P. R. Badovinatz et al., Ser. No. 08/641,445;

"Method For Group Leader Recovery In A Distributed ComputingEnvironment," by P. R. Badovinatz et al., U.S. Pat. No. 5,704,032; and

"A Method For Barrier Synchronization In A Distributed ComputingEnvironment," by P. R. Badovinatz et al., Ser. No. 08/640,218.

BACKGROUND ART

In many data processing systems, messages are broadcast to all of theprocessors of the system. That is, there exists a single message stream,in which all of the nodes receive all of the messages. This all to allcommunication places a practical limit on the number of processors thatcan be a part of the system.

Therefore, a need exists for a communications mechanism in whichmessages can be forwarded to particular groups of processors within thesystem. A further need exists for a mechanism that permits one set ofmessages to be sent to one group of processsors, while another set ofmessages is sent to another group of processors, and each groupmaintains its own set of messages. A further need exists for a mechanismthat orders the messages in a particular sequence, such that a processorknows if it is missing a particular message. Additionally, a need existsfor a mechanism that enables a processor to obtain any missing message,without needing to obtain the missing message from persistent storage. Afurther need exists for a mechanism that can quiesce a multicast stream.

SUMMARY OF THE INVENTION

The shortcomings of the prior art are overcome and additional advantagesare provided through the provision of a communications mechanism for adistributed computing environment. A first message is sent to a firstgroup of processors of the distributed computing environment and asecond message is sent to a second group of processors. The first groupof processors maintains the first message and the second group ofprocessors maintains the second message.

In one embodiment of the invention, a request is sent to a group leaderof the first group of processors to multicast the first message. Thegroup leader assigns a sequence number to the first message providing afirst ordered message. Then, the group leader multicasts the firstordered message to the first group of processors. Each processor of thefirst group of processors determines whether the first ordered messageit received is in proper sequence order. If a processor determines thatthe ordered message is in improper sequence order indicating that amessage is missing, then that processor requests the missing messagefrom a processor within the group.

The communications mechanism of the present invention advantageouslyeliminates the need for separate stable storage typically used when afailure occurs within the system. Instead, the mechanism of the presentinvention, replicates key parts of data in a recoverable fashion acrossthe processors that may need that data. Individual processors areadvantageously aware that they are missing data and are provided amechanism for retrieving that data from any other processor within itsgroup that has the data. That is, information that is needed by anyindividual group is replicated across all of the processors of thatgroup and thus, it can recover that data without needing or relying onseparate, stable hardware. This reduces the costs associated withadditional hardware and provides for a flexible system.

The communications mechanism of the present invention alsoadvantageously enables quiescing of a multicast stream.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention will be apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one example of a distributed computing environmentincorporating the principles of the present invention;

FIG. 2 depicts one example of an expanded view of a number of theprocessing nodes of the distributed computing environment of FIG. 1, inaccordance with the principles of the present invention;

FIG. 3 depicts one example of the components of a Group Servicesfacility, in accordance with the principles of the present invention;

FIG. 4 illustrates one example of a processor group, in accordance withthe principles of the present invention;

FIG. 5a depicts one example of the logic associated with recovering froma failed group leader of the processor group of FIG. 4, in accordancewith the principles of the present invention;

FIG. 5b depicts another example of the logic associated with recoveringfrom a failed group leader of the processor group of FIG. 4, inaccordance with the principles of the present invention;

FIG. 6a illustrates one example of a group leader, in accordance withthe principles of the present invention;

FIG. 6b illustrates a technique for selecting a new group leader whenthe current group leader fails, in accordance with the principles of thepresent invention;

FIG. 7 depicts one example of a name server receiving information from agroup leader, in accordance with the principles of the presentinvention;

FIG. 8 depicts one example of the logic associated with adding aprocessor to a group of processors, in accordance with the principles ofthe present invention;

FIG. 9 depicts one example of the logic associated with a processorleaving a group of processors, in accordance with the principles of thepresent invention;

FIG. 10 illustrates one embodiment of a process group, in accordancewith the principles of the present invention;

FIG. 11 depicts one example of the logic associated with proposing aprotocol for a process group, in accordance with the principles of thepresent invention;

FIG. 12 depicts one example of the logic associated with a processrequesting to join a process group, in accordance with the principles ofthe present invention; and

FIG. 13 depicts one example of the logic associated with a member of aprocess group requesting to leave the group, in accordance with theprinciples of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

In one embodiment, the techniques of the present invention are used indistributed computing environments in order to provide multicomputerapplications that are highly-available. Applications that arehighly-available are able to continue to execute after a failure. Thatis, the application is fault-tolerant and the integrity of customer datais preserved.

It is important in highly-available systems to be able to coordinate,manage and monitor changes to subsystems (e.g., process groups) runningon processing nodes within the distributed computing environment. Inaccordance with the principles of the present invention, a facility isprovided that implements the above functions. One example of such afacility is referred to herein as Group Services.

Group Services is a system-wide, fault-tolerant and highly-availableservice that provides a facility for coordinating, managing andmonitoring changes to a subsystem running on one or more processors of adistributed computing environment. Group Services, through thetechniques of the present invention, provides an integrated frameworkfor designing and implementing fault-tolerant subsystems and forproviding consistent recovery of multiple subsystems. Group Servicesoffers a simple programming model based on a small number of coreconcepts. These concepts include, in accordance with the principles ofthe present invention, a clusterwide process group membership andsynchronization service that maintains application specific informationwith each process group.

As described above, in one example, the mechanisms of the presentinvention are included in a Group Services facility. However, themechanisms of the present invention can be used in or with various otherfacilities, and thus, Group Services is only one example. The use of theterm Group Services to include the techniques of the present inventionis for convenience only.

In one embodiment, the mechanisms of the present invention areincorporated and used in a distributed computing environment, such asthe one depicted in FIG. 1. In one example, distributed computingenvironment 100 includes, for instance, a plurality of frames 102coupled to one another via a plurality of LAN gates 104. Frames 102 andLAN gates 104 are described in detail below.

In one example, distributed computing environment 100 includes eight (8)frames, each of which includes a plurality of processing nodes 106. Inone instance, each frame includes sixteen (16) processing nodes (a.k.a,processors). Each processing node is, for instance, a RISC/6000 computerrunning AIX, a UNIX based operating system. Each processing node withina frame is coupled to the other processing nodes of the frame via, forexample, an internal LAN connection. Additionally, each frame is coupledto the other frames via LAN gates 104.

As examples, each LAN gate 104 includes either a RISC/6000 computer, anycomputer network connection to the LAN, or a network router. However,these are only examples. It will be apparent to those skilled in therelevant art that there are other types of LAN gates, and that othermechanisms can also be used to couple the frames to one another.

In addition to the above, the distributed computing environment of FIG.1 is only one example. It is possible to have more or less than eightframes, or more or less than sixteen nodes per frame. Further, theprocessing nodes do not have to be RISC/6000 computers running AIX. Someor all of the processing nodes can include different types of computersand/or different operating systems. All of these variations areconsidered a part of the claimed invention.

In one embodiment, a Group Services subsystem incorporating themechanisms of the present invention is distributed across a plurality ofthe processing nodes of distributed computing environment 100. Inparticular, in one example, a Group Services daemon 200 (FIG. 2) islocated within one or more of processing nodes 106. The Group Servicesdaemons are collectively referred to as Group Services.

Group Services facilitates, for instance, communication andsynchronization between multiple processes of a process group, and canbe used in a variety of situations, including, for example, providing adistributed recovery synchronization mechanism. A process 202 (FIG. 2)desirous of using the facilities of Group Services is coupled to a GroupServices daemon 200. In particular, the process is coupled to GroupServices by linking at least a part of the code associated with GroupServices (e.g., the library code) into its own code. In accordance withthe principles of the present invention, this linkage enables theprocess to use the mechanisms of the present invention, as described indetail below.

In one embodiment, a process uses the mechanisms of the presentinvention via an application programming interface 204. In particular,the application programming interface provides an interface for theprocess to use the mechanisms of the present invention, which areincluded in Group Services, as one example. In one embodiment, GroupServices 200 includes an internal layer 302 (FIG. 3) and an externallayer 304, each of which is described in detail below.

In accordance with the principles of the present invention, internallayer 302 provides a limited set of functions for external layer 304.The limited set of functions of the internal layer can be used to builda richer and broader set of functions, which are implemented by theexternal layer and exported to the processes via the applicationprogramming interface. The internal layer of Group Services (alsoreferred to as a metagroup layer) is concerned with the Group Servicesdaemons, and not the processes (i.e., the client processes) coupled tothe daemons. That is, the internal layer focuses its efforts on theprocessors, which include the daemons. In one example, there is only oneGroup Services daemon on a processing node; however, a subset or all ofthe processing nodes within the distributed computing environment caninclude Group Services daemons.

The internal layer of Group Services implements functions on a perprocessor group basis. There may be a plurality of processor groups inthe network. Each processor group (also, referred to as a metagroup)includes one or more processors having a Group Services daemon executingthereon. The processors of a particular group are related in that theyare executing related processes. (In one example, processes that arerelated provide a common function.) For example, referring to FIG. 4, aProcessor Group X (400) includes Processing Node 1 and Processing Node2, since each of these nodes is executing a process X, but it does notinclude Processing Node 3. Thus, Processing Nodes 1 and 2 are members ofProcessor Group X. A processing node can be a member of none or anynumber of processor groups, and processor groups can have one or moremembers in common.

In order to become a member of a processor group, a processor needs torequest to be a member of that group. In accordance with the principlesof the present invention, a processor requests to become a member of aparticular processor group (e.g., Processor Group X) when a processrelated to that group (e.g., Process X) requests to join a correspondingprocess group (e.g., Process Group X) and the processor is not aware ofthat corresponding process group. Since the Group Services daemon on theprocessor handling the request to join a particular process group is notaware of the process group, it knows that it is not a member of thecorresponding processor group. Thus, the processor asks to become amember, so that the process can become a member of the process group.(One technique for becoming a member of a processor group is describedin detail further below.)

Internal layer 302 (FIG. 3) implements a number of functions on a perprocessor group basis. These functions include, for example, maintenanceof group leaders, insert, multicast, leave, and fail, each of which isdescribed in detail below.

In accordance with the principles of the present invention, a groupleader is selected for each processor group of the network. In oneexample, the group leader is the first processor requesting to join aparticular group. As described herein, the group leader is responsiblefor controlling activities associated with its processor group(s). Forexample, if a processing node, Node 2 (FIG. 4), is the first node torequest to join Processor Group X, then Processing Node 2 is the groupleader and is responsible for managing the activities of Processor GroupX. It is possible for Processing Node 2 to be the group leader ofmultiple processor groups.

If the group leader is removed from the processor group for any reason,including, for instance, the processor requests to leave the group, theprocessor fails or the Group Services daemon on the processor fails,then group leader recovery takes place. In particular, a new groupleader is selected, STEP 500a "SELECT NEW GROUP LEADER" (FIG. 5a).

In one example, in order to select a new group leader, a membership listfor the processor group, which is ordered in sequence of processorsjoining the group, is scanned, by one or more processors of the group,for the next processor in the list, STEP 502 "OBTAIN NEXT MEMBER INMEMBERSHIP LIST." Thereafter, a determination is made as to whether theprocessor obtained from the list is active, INQUIRY 504 "IS MEMBERACTIVE?" In one embodiment, this is determined by another subsystemdistributed across the processing nodes of the distributed computingenvironment. The subsystem sends a signal to at least the nodes in themembership list, and if there is no response from a particular node, itassumes the node is inactive.

If the selected processor is not active, then the membership list isscanned, again until an active member is located. When an activeprocessor is obtained from the list, then this processor is the newgroup leader for the processor group, STEP 506 "SELECTED MEMBER IS NEWGROUP LEADER."

For example, assume that three processing nodes joined Processor Group Xin the following order:

Processor 2, Processor 1, and Processor 3.

Thus, Processor 2 is the initial group leader (see FIG. 6a). At sometime later, Processor 2 leaves Processor Group X, and therefore, a newgroup leader is desired. According to the membership list for ProcessorGroup X, Processor 1 is the next group leader. However, if Processor 1is inactive, then Processor 3 would be chosen to be the new group leader(FIG. 6b).

In accordance with the principles of the present invention, in oneexample, the membership list is stored in memory of each of theprocessing nodes of the processor group. Thus, in the above example,Processor 1, Processor 2, and Processor 3 would all contain a copy ofthe membership list. In particular, each processor to join the groupreceives a copy of the membership list from the current group leader. Inanother example, each processor to join the group receives themembership list from another member of the group other than the currentgroup leader.

Referring back to FIG. 5a, in one embodiment of the invention, once thenew group leader is selected, the new group leader informs a name serverthat it is the new group leader, STEP 508 "INFORM NAME SERVER." As oneexample, a name server 700 (FIG. 7) is one of the processing nodeswithin the distributed computing environment designated to be the nameserver. The name server serves as a central location for storing certaininformation, including, for instance, a list of all of the processorgroups of the network and a list of the group leaders for all of theprocessor groups. This information is stored in the memory of the nameserver processing node. The name server can be a processing node withinthe processor group or a processing node independent of the processorgroup.

In one example, name server 700 is informed of the group leader changevia a message sent from the Group Services daemon of the new groupleader to the name server. Thereafter, the name server then informs theother processors of the group of the new group leader via, for example,an atomic multicast, STEP 510 "INFORM OTHER MEMBERS OF THE GROUP" (FIG.5a). (Multicasting is similar in function to broadcasting, however, inmulticasting the message is directed to a selected group, instead ofbeing provided to all processors of a system. In one example,multicasting can be performed by providing software that takes themessage and the list of intended recipients and performs point to pointmessaging to each intended recipient using, for example, a User DatagramProtocol (UDP) or a Transmission Control Protocol (TCP). In anotherembodiment, the message and list of intended recipients are passed tothe underlying hardware communications, such as Ethernet, which willprovide the multicasting function.)

In another embodiment of the invention, a member of the group other thanthe new group leader informs the name server of the identity of the newgroup leader. As a further example, the processors of the group are notexplicitly informed of the new group leader, since each processor in theprocessor group has the membership list and has determined for itselfthe new group leader.

In yet another embodiment of the invention, when a new group leader isneeded, a request is sent to the name server requesting from the nameserver the identity of the new group leader, STEP 500b "REQUEST NEWGROUP LEADER FROM NAME SERVER" (FIG. 5b). In this embodiment, themembership list is also located at the name server, and the name servergoes through the same steps described above for determining the newgroup leader, STEPS 502, 504 and 506. Once it is determined, the nameserver informs the other processors of the processor group of the newgroup leader, STEP 510 "INFORM OTHER MEMBERS OF THE GROUP."

In addition to the group leader maintenance function implemented by theinternal or metagroup layer, an insert function is also implemented. Theinsert function is used when a Group Services daemon (i.e., a processorexecuting the Group Services daemon) wishes to join a particular groupof processors. As described above, a processor requests to be added to aparticular processor group when a process executing on the processorwishes to join a process group and the processor is unaware of theprocess group.

In one example, in order to become a member of a processor group, theprocessor wishing to join the group first determines who is the groupleader of the processor group, STEP 800 "DETERMINE GROUP LEADER" (FIG.8). In one embodiment, the group leader is determined by providing nameserver 700 with the name of the processor group and requesting from thename server the identity of the group leader for that group.

Should the name server respond that the requesting processor is thegroup leader (since this is the first request for the group), INQUIRY801, the requesting processor forms the processor group, STEP 803 "FORMGROUP." In particular, it creates a membership list for that particularprocessor group, which includes the requesting processor.

If the processor is not the group leader, then it sends an insertrequest, via a message, to the group leader, the identity of which isobtained from the name server, STEP 802 "SEND INSERT REQUEST TO GROUPLEADER." The group leader then adds the requesting processor to theprocessor group, STEP 804 "GROUP LEADER INSERTS PROCESSOR IN PROCESSORGROUP." In particular, in one embodiment, the Group Services daemon ofthe group leader updates its membership list and informs, via amulticast, each other Group Services daemon of the processor group toadd the joining processor to the membership list located at thatprocessor. In particular, as one example, the group leader informs theother daemons, via a multicast, of the update, the daemons acknowledgethe update, and then the group leader sends out a commit for the changevia another multicast. (In another embodiment, the informing can beperformed via an atomic multicast.) In one example, the joiningprocessor is added to the end of the membership list, since the list ismaintained by order of joins to the group.

In accordance with the principles of the present invention, a processorthat is a member of a processor group may request to leave the group.Similar to the insert request, a leave request is forwarded to the groupleader via, for instance, a message, STEP 900 "SEND LEAVE REQUEST TOGROUP LEADER" (FIG. 9). Thereafter, the group leader removes theprocessor from the group by, for example, deleting the processor fromits membership list and informing all members of the processor group toalso remove the processor from their membership list, STEP 902 "GROUPLEADER REMOVES PROCESSOR FROM GROUP." Additionally, if the leavingprocessor is the group leader, then group leader recovery takes place,as described above.

In addition to the above, if a processor fails, or if the Group Servicesdaemon executing on the processor fails, the processor is removed fromthe processor group. In one embodiment, when the Group Services daemonfails, it is assumed that the processor fails. In one example, a failedprocessor is detected by a subsystem running within the distributedcomputing environment that detects processor failure. When there is afailure, in one instance, the processor is removed by the group leader.In particular, the group leader deletes the processor from itsmembership list and informs the other member processors to do the same,as described above.

Another function implemented by the internal layer of Group Services isa multicast function. In accordance with the principles of the presentinvention, a member of a processor group can multicast a message to theother members of the group. This multicast can include one-waymulticasts, as well as acknowledged multicasts.

In one embodiment, in order to multicast a message from one member of agroup to other members of the group, the message sending member sendsthe message to the group leader of the group, and the group leadermulticasts the message to the other members.

In accordance with the principles of the present invention, prior tosending a message, the group leader assigns a sequence number to themessage. Assigned sequence numbers are kept in numerical order. Thus, ifa member of the processor group (i.e., Group Services) receives amessage having a sequence number out of order, it knows that it hasmissed a message. For instance, if a processing node receives messages43 and 45, it knows it missed message 44.

In accordance with the principles of the present invention, theprocessing node can retrieve the missing message from any of theprocessing nodes in the processor group, since all of the nodes in thegroup have received the same messages. However, in one example, theprocessing node missing the information requests it from the groupleader. However, if it is the group leader that is missing the message,then it can request it from any of the other processing nodes in theprocessor group. This is possible since key data is replicated acrossall of the processing nodes of the processor group, in a recoverablefashion. There is no need, in accordance with the present invention, tostore the data required for recovery in persistent storage. Thetechnique of the present invention eliminates the need for persistentstable hardware-based storage for storing recovery data.

If, for example, the group leader fails, a new group leader is selected,as described above. The group leader ensures that it has all of themessages by communicating with the processing nodes of the group. In oneembodiment, once the group leader is sure that it has all of themessages, it ensures that all of the other processing nodes of the groupalso have those messages. The technique of the present invention thus,allows recovery from a failed processing node, failed processes, or linkwithout requiring stable storage.

In accordance with the principles of the present invention, eachprocessor group maintains its own ordered set of messages. Thus, themessages for one processor group will not overlap or interfere with themessages of another processor group. The processor groups, along withtheir ordered messages, are independent of one another. Therefore, oneprocessor group may receive an ordered set of messages of 43, 44 and 45,while another processor group may receive an independently ordered setof messages of 1, 2 and 3. This avoids the need for all to allcommunication among all of the processors of a network.

In one embodiment of the invention, each processing node retains themessages it receives for a certain amount of time, in case it needs toprovide the message to another node or in case it becomes the groupleader. The messages are saved until the messages are received by all ofthe processors of the group. Once the messages are received by all ofthe processors, then the messages can be discarded.

In one example, it is the group leader that informs the processing nodesthat the messages have been received by all of the nodes. Specifically,in one example, when a processing node sends a message to the groupleader, it includes an indication of the last message that it has seen(i.e., the last message in proper order). The group leader collects thisinformation, and when it sends a message to the processing nodes, itincludes in the message the sequence number of the last message seen byall of the nodes. Thereafter, the processing nodes can delete thosemessages indicated as being seen.

In accordance with the principles of the present invention, themulticast stream is advantageously quiesced at certain times to insureall processor group members have received all of the messages. Forexample, the stream is quiesced when there have been no multicasts for acertain period of time or after some number of NoAckRequired (i.e., noacknowledgment required) multicasts have been sent. In one embodiment,when the multicast stream is to be quiesced, the group leader sends outa SYNC multicast, which all processor group members acknowledge. When aprocessor group member receives such a message, it knows that it has (orshould have) all of the messages, based on the sequence number of theSYNC message. If it is missing any messages, it obtains the messagesbefore acknowledging. When the group leader receives all of theacknowledgments to this multicast, it knows that all processor groupmembers have received all of the messages, and therefore, the multicaststream is synced and quiesced.

In another embodiment of the invention, a specific SYNC multicast is notnecessary. Instead, one of the following techniques can be used toquiesce the multicast stream. As one example, a multicast requiring anacknowledgment can be sent from the group leader to the processors. Whena processor receives a multicast that requires an acknowledgment, itsends the acknowledgment to the group leader. The acknowledgmentcontains the sequence number of the multicast it is acknowledging. Theprocessors use this sequence number to determine if they are missing anymessages. If so, they request the missing messages from the groupleader, as one example. After the group leader multicasts theACK-required message to all of the processors of the group and receivesall of the acknowledgments, the group leader knows that the stream isquiesced. The non-group leader processors rely on the group leader toinsure that they receive all the messages in a timely fashion, so theydo not need to periodically acknowledge or ping the group leader toinsure they have not missed a multicast.

As a further example, in those situations in which NoAckRequiredmulticasts are being used, the group leader can alter one of theNoAckRequired multicasts into an AckRequired multicast, thus using it asa sync in the manner described above. Thus, no explicit SYNC message isrequired.

In addition to the above, in another example, it is possible for thenon-group leader processors to anticipate the group leader's action,such that if the number of NoAckRequired messages approaches the windowsize (i.e., e.g., reaches a predetermined number, such as five, in oneexample) or if a maximum idle time approaches, the non-group leaderprocessors can send an ACK to the group leader. The ACK provides to thegroup leader the highest sequence number multicast that each processorhas received. If all of the non-group leader processors do this, then itis not necessary for the group leader to turn a NoAckRequired multicastinto an AckRequired multicast. Therefore, the group is not held up bywaiting for all of the acknowledgments.

Support for the above feature of the present invention is transparent tothe users of Group Services (i.e., the processes). No explicit actionsare necessary by the processes to implement this feature. Additionally,this support is available in the internal and external layers of GroupServices.

Referring back to FIG. 3, external layer 304 implements a richer set ofmechanisms of the application programming interface that is easy for theuser (i.e., the client processes) to understand.

In one example, these mechanisms include an atomic multicast, a 2-phasecommit, barrier synchronization, process group membership, processorgroup membership, and process group state value, each of which isdescribed below. These mechanisms, as well as others, are unified, inaccordance with the principles of the present invention, by theapplication programming interface, into a single, unified framework thatis easy to understand. In particular, communications and synchronizationmechanisms (in addition to other mechanisms) have been unified into asingle protocol.

In accordance with the principles of the present invention, the single,unified framework is provided to members of process groups, as describedin detail herein. A process group includes one or more related processesexecuting on one or more processing nodes of the distributed computingenvironment. For example, referring to FIG. 10, a Process Group X (1000)includes a Process X executing on Processor 1 and two Process X'sexecuting on Processor 2. The manner in which a process becomes a memberof a particular process group is described in detail further below.

Process groups can have at least two types of members, including aprovider and a subscriber. A provider is a member process that hascertain privileges, such as voting rights, and a subscriber has no suchprivileges. A subscriber can merely watch the ongoings of a processgroup, but cannot participate in the group. For example, a subscribercan monitor the membership of a group, as well as the state value of thegroup, but it cannot vote. In other embodiments, other types of memberswith differing rights can be provided.

In accordance with the principles of the present invention, theapplication programming interface is implemented, as described belowwith reference to FIG. 11.

Referring to FIG. 11, in one example, initially, a provider of a processgroup proposes a protocol for the group (subscribers cannot proposeprotocols, in this embodiment), STEP 1100 "MEMBER OF PROCESS GROUPPROPOSES A PROTOCOL FOR THE GROUP." In particular, in one instance, anAPI call is made proposing the protocol. In one example, the protocol issubmitted, by a process, to the external layer of the Group Servicesdaemon on the processor executing the process. That Group Servicesdaemon then submits the protocol to the group leader of the group via amessage. The group leader then informs, via a multicast, all of theprocessors of the related processor group of the protocol. (The internallayer of the daemon is managing this multicast.) Those processors theninform the appropriate members of the process group, via the externallayer, of the proposed protocol, STEP 1102 "INFORM PROCESS GROUP MEMBERSOF THE PROTOCOL."

If multiple providers propose a protocol at the same time, then thegroup leader selects the protocol to be run, in the following manner. Inone embodiment, the protocols are prioritized in that any protocol for afailure is first, a join protocol is second, and all other protocols(e.g., requests to leave, expel, update state value and provide a groupmessage, described below) are on a first come first served basis. Thus,if a request to remove a member due to a failure is proposed at the sametime as a request to join and a request to leave, then the request toremove is selected first. Then, the request to join is selected,followed by the request to leave.

If there are multiple requests to remove due to failure, then all ofthese requests are selected prior to the request to join. The requeststo remove are selected by the group leader in the order seen by thegroup leader (unless batching is allowed, as described below).Similarly, if there are multiple request to join, then these areselected in a likewise manner prior to any of the other requests.

In one embodiment, if there are multiple other requests, the first onereceived by the group leader is selected and the others are dropped. Thegroup leader informs the providers of those dropped requests that theyhave been dropped and then, they can resubmit them if they wish. Inanother embodiment of the invention, these other requests can be queuedin order of receipt and selected in turn, instead of being dropped.

After a protocol is selected, a determination is made as to whethervoting should be performed for the protocol, INQUIRY 1104 "VOTING?" Inone example, the process proposing the protocol indicates during theinitial proposal whether voting is to take place. If the proposalindicates no voting, then the protocol is simply an atomic multicast,and the protocol is complete, STEP 1106 "END."

If voting is to take place, then each provider of the process groupvotes on the protocol, STEP 1108 "PROCESS GROUP MEMBERS WITH VOTINGPRIVILEGES VOTE." Specifically, in accordance with the principles of thepresent invention, the voting allows each provider to take local actionsnecessary to satisfy the group, and to inform the group of the resultsof those actions. This functions as a barrier synchronization primitiveby ensuring that all providers have reached a particular point beforeproceeding.

In one embodiment of the present invention, each provider votes bycasting a vote value, which may include one of the following, as anexample:

(a) APPROVE specifying that the provider wishes to complete the protocolonce all of the providers have reached this barrier, and to accept allthe proposed changes;

(b) CONTINUE specifying that the provider wishes to continue theprotocol through another voting step, and proposed changes remainpending; and

(c) REJECT specifying that the provider wishes to end this protocol onceall the providers have reached this barrier, and to reject thoseproposed changes that can be rejected.

In accordance with the principles of the present invention, eachprovider of the process group forwards its vote to the Group Servicesdaemon executing on the same processor as the process. The GroupServices daemon then forwards the vote values it receives to the groupleader for the metagroup associated with that process group. Forinstance, the vote values for Process Group X are forwarded to the groupleader of Processor Group X. Based on the vote values, the group leaderdetermines how the protocol should proceed. The group leader thenmulticasts the result of the voting to each of the processors of theappropriate processor group (i.e., to the Group Services daemons onthose processors), and the Group Services daemons inform the providersof the result value. For example, the group leader informs the GroupServices daemons of Processor Group X and the Group Services daemonsprovide the result to the providers of Process Group X.

If one of the providers voted CONTINUE and none of the providers votedREJECT, INQUIRY 1110 "CONTINUE VOTING?", then the protocol proceeds toanother voting step, STEP 1108. That is, the providers are performingbarrier synchronization with a dynamic number of synchronization phases.In particular, in accordance with the principles of the presentinvention, the number of voting steps (or synchronization phases orpoints) that a protocol can have is dynamic. It can be any number ofsteps desired by the voting members. The protocol can continue as longas any provider wishes for the protocol to continue. Thus, in oneembodiment, the voting dynamically controls the number of voting steps.However, in another embodiment, the dynamic number of voting steps canbe set during the initiation of the protocol. It is still dynamic, sinceit can change each time the protocol is initialized.

If the providers vote not to continue to another voting step, then theprotocol is a 2-phase commit. After the voting is complete (either for atwo-phase or multi-phase vote), the result of the vote is provided tothe members. In particular, should any one provider of the process groupvote REJECT, then the protocol ends and the proposed changes arerejected. Each of the providers is informed, via a multicast, that theprotocol has been rejected, STEP 1112 "INFORM MEMBERS OF COMPLETION OFPROTOCOL." On the other hand, if all of the providers voted APPROVE,then the protocol is complete and all of the proposed changes areaccepted. The providers are informed of the approved protocol, via amulticast, STEP 1112 "INFORM MEMBERS OF COMPLETION OF PROTOCOL."

In accordance with the principles of the present invention, theabove-described protocol is also integrated with process groupmembership and process group state values. In particular, the mechanismsof the present invention are used to manage and monitor membershipchanges to the process groups. Changes to group membership are proposedvia the protocol described above. Additionally, the mechanisms of thepresent invention mediate changes to the group state value, andguarantee that it remains consistent and reliable, as long as at leastone process group member remains.

A group state value for the process group acts as a synchronizedblackboard for the process group. In one embodiment, the group statevalue is an application specific value controlled by the providers. Thegroup state value is part of the group state data maintained for eachprocess group by Group Services. In addition to the group state value,the group state data includes a provider membership list for that group.Each provider is identified by a provider identifier and the list isordered by Group Services such that the oldest provider (the firstprovider joining the group) is at the head of the list, and the youngestis at the end.

Changes to the group state value are proposed by group members (i.e.,the providers) via the protocol described above. In one embodiment, thecontents of the group state value are not interpreted by Group Services.The meaning of the group state value is attached by the group members.The mechanisms of the present invention guarantee that all process groupmembers see the same sequence of changes to the group state values, andthat all process group members will see the updates.

Thus, as described above, the application programming interface of thepresent invention provides a single, unified protocol that includes aplurality of mechanisms including, for example, an atomic multicast,2-phase commit, barrier synchronization, group membership and groupstate value. The manner in which the protocol is used for groupmembership and the group state value is described in further detailbelow.

The voting mechanism described above is used, in accordance with theprinciples of the present invention, to propose changes to themembership of a process group. For instance, if a process wishes to joina particular process group, such as Process Group X, then that processissues a join call, STEP 1200 "INITIATE REQUEST TO JOIN" (FIG. 12). Inone embodiment, this call is sent as a message across a localcommunications path (e.g., a UNIX domain socket) to the Group Servicesdaemon on the processor executing the requesting process. The GroupServices daemon sends a message to the name server asking the nameserver for the name of the group leader for the process group that therequesting process wishes to join, STEP 1202 "DETERMINE GROUP LEADER."

If this is the first request to join the particular process group, thenthe name server informs the Group Services daemon that it is the groupleader, INQUIRY 1204 "FIRST REQUEST TO JOIN?". Thus, the processorcreates a processor group, as described above, and adds the process tothe process group, STEP 1210 "ADD PROCESS." In particular, the processis added to a membership list for that process group. This membershiplist is maintained by Group Services, for example, as an ordered list.In one example, it is ordered in sequence of joins. The first process tojoin is first in the list, and so forth.

In accordance with the principles of the present invention, the firstprocess to join a process group identifies a set of attributes for thegroup. These attributes are included as arguments in the join call sentby the process. These attributes include, for instance, the group name,which is a unique identifier, and prespecified information that definesto Group Services how the group wishes to manage various protocols. Forinstance, the attributes can include an indication of whether theprocess group will accept batched requests, as described below.Additionally, in another example, the attributes can include a clientversion number representing, for example, the software level of theprogramming in each provider. This will ensure that all group membersare at the same level. The above-described attributes are only oneexample. Additional or different attributes can be included withoutdeparting from the spirit of the claimed invention.

Returning to INQUIRY 1204 "FIRST REQUEST TO JOIN?", if this is not thefirst request to join, then the join request is sent via a message tothe group leader, designated by the name server, STEP 1214 "SEND JOINREQUEST TO GROUP LEADER." The group leader then performs a prescreeningtest, STEP 1216 "PRESCREEN." In particular, the group leader determineswhether the attributes specified by the requesting process are the sameas the attributes set by the first process of the group. If not, thenthe join request is rejected.

However, if the prescreen test is successful, then the providers of theprocess group are informed of the request via, for instance, a multicastfrom the group leader, and the providers vote on whether to allow theprocess to be added to the group, STEP 1220 "VOTE." The voting takesplace, as described above. The providers can vote to continue theprotocol and vote on this join again, or they can vote to reject orapprove the join. If one of the providers votes REJECT, then the join isterminated and the process is not added to the group, INQUIRY 1222"SUCCESSFUL?". However, if all of the providers vote APPROVE, then theprocess is added to the group, STEP 1224 "ADD PROCESS." In particular,the process is added to the end of the membership list for the group.Once the protocol is complete, the members of the group are notified ofthe result. In particular, in one example, all of the members (includingthe providers and subscribers) are notified when the process is added,but only the providers are notified when the protocol has been rejected.In another example, other types of members may also be notified, asdeemed appropriate.

Join requests are used by providers to join a process group, asdescribed above. A provider is afforded certain benefits, such as votingrights. Processes can also subscribe to a process group, however, byissuing an API subscribe call (as opposed to a join call). A subscriberis provided the ability to monitor a particular process group, but notto participate in the group.

When a subscribe call is issued, it is forwarded to the Group Servicesdaemon on that processor and that Group Services daemon keeps track ofit. If the Group Services daemon is not a part of the processor group,then it will become inserted into the group, as previously described. Inone embodiment, there is no voting for the subscriber, and other membersof the group, including the providers and any other subscribers, are notaware of the subscriber. A subscriber cannot subscribe to a processgroup that is not already created.

Group membership can also be altered by a group member leaving or beingremoved from a group. In one example, a group member wishing to leave agroup, sends a request to leave to the group leader, in the mannerdescribed above, STEP 1300 "INITIATE REQUEST TO LEAVE" (FIG. 13). Thegroup leader sends a multicast to the providers requesting the providersto vote on the proposed change, STEP 1302 "VOTE." The vote takes placein the manner described above, and if all of the providers vote APPROVE,INQUIRY 1304, then the process is removed from the membership list forthat process group, STEP 1306 "REMOVE PROCESS," and all of the groupmembers are notified of the change. However, if one of the providersvotes REJECT, then the process remains a part of the process group, theprotocol is terminated, and the providers are notified of the rejectedprotocol. Of course, if none of the providers votes REJECT and any oneof the providers votes CONTINUE, then the protocol continues to anotherround of voting.

A member of a group may leave the group involuntarily when it isexpelled from the group via an approved expel protocol proposed byanother process of the group, or when the group member fails or theprocessor in which it is executing fails. The manner in which anexpulsion is performed is the same as that described above for a memberrequesting to leave a group, except that the request is not initiated bya process wishing to leave, but instead by a process desiring to removeanother process from the group.

Likewise, in one embodiment, the technique for removing a process whenthe process fails or when the processor executing the process fails, issimilar to that technique used to remove a process requesting to leave.However, instead of the process initiating a request to leave, therequest is initiated by Group Services, as described below.

In the case of a process failure, in one example, the group leader isinformed of the failure by the Group Services daemon running on theprocessor of the failed process. The Group Services daemon determinesthat the process has failed, when it detects that a stream socket (knownto those skilled in the art) associated with the process has failed. Thegroup leader then initiates the removal.

In the case of a processor failure, the group leader detects thisfailure and initiates the request to remove. If it is the group leaderthat has failed, then group leader recovery is performed, as describedherein, before the request is initiated. In one embodiment, the groupleader is informed of the processor failure by a subsystem that isdistributed across the processing nodes of the network. This subsystemsends out signals to all of the processing nodes and if the signal isnot acknowledged by a particular node, that node is considered down (orfailed). This information is then broadcast to Group Services.

As described above, when a process wishes to join a group or a groupmember wishes to leave or is removed from the group, the group leaderinforms each of the group providers of the proposed change, so that theproviders can vote on that change. In accordance with the principles ofthe present invention, these proposed membership changes can bepresented to the group providers either singly (i.e., one proposed groupmembership change per protocol) or batched (i.e., multiple proposedgroup membership changes per protocol). In the case of batched requests,the group leader collects the requests for a prespecified amount oftime, as one example, and then presents to the group providers one ormore batched requests. Specifically, one batched request is provided,which includes all of the join requests collected during that time, andanother batched request is provided, which includes all of the leave orremove requests collected. In one embodiment, one batched request canonly include all joins or all leaves (and removals), and not acombination of both. This is only one example. In other examples, it ispossible to combine both types of requests.

When a batched request is forwarded to the group providers, the groupproviders vote on the entire batched request, as a whole. Thus, eitherthe entire batch is accepted, continued or rejected.

In accordance with the principles of the present invention, each processgroup can determine whether it is willing to allow requests to bebatched or not. Additionally, each process group can determine whethersome types of requests are allowed to be batched, while others are not.For instance, assume there are a number of process groups executing inthe network. Process Group W can decide that it wants to receive batchedrequests for all types of requests, while Process Group X canindependently decide that it wants to receive all requests serially.Additionally, Process Group Y can allow batched request for only joinrequests, while Process Group Z allows batched requests only for leaveor removal requests. Thus, the mechanisms of the present inventionprovide flexibility in how requests are presented and voted on.

Although the system is flexible, there are a number of rules that havebeen instituted in one embodiment of the invention to ensure consistentand reliable group membership. These rules include the following, as oneexample:

1. No group member can be shown to be failing and leaving the groupbefore it has joined the group.

2. No group member can be shown to be joining a group a second time,before its initial failure has been handled.

3. Where a group has both requests to join, and has established membersin a failed state, all of the failed members are dealt with (via one ormore of the failure protocols) before any of the requests to join can besatisfied.

4. All non-failed group providers, including those requesting to join,see the same sequence of protocols and membership lists.

Described above in detail is how the voting protocol of the presentinvention is used to manage group membership. The voting protocol canalso be used, however, to propose a group state value, in accordancewith the principles of the present invention. In particular, during avoting phase, a provider of the process group can propose to change thestate value of the group, in addition to providing a vote value. Thisprovides a mechanism to allow group providers to reflect groupinformation reliably and consistently to other group members. In oneexample, the group state value (and other information, such as, amessage, and an updated vote value, as described herein) is providedwith the vote value via a vote interface that allows for variousarguments to be presented.

For example, when a member joins or leaves the group, the group isdriven through a multi-step protocol, as described above. During eachvoting step, the group members perform local actions to prepare for thenew member, or to recover from the loss of the failed member. Based onthe results of these local actions, for instance, one or more of theproviders may decide to modify the group state value. In one example,the group state value can be "active," indicating that the process groupis ready to accept service requests; "inactive," indicating that theprocess group is shutdown because, for instance, the group does not haveenough members; or "suspend," indicating that the process group willaccept requests, but is temporarily not processing the requests.

Group Services guarantees that the updates to the group state value arecoordinated, such that the group providers will see the same consistentvalue. If the protocol is APPROVED, then the latest updated proposedgroup state value is the new group state value. If the protocol isREJECTED, then the group's state value remains as it was before therejected protocol began execution.

In accordance with the principles of the present invention, the votingprotocol can also be used to multicast messages to the group members.For example, in addition to providing a vote value, a provider caninclude a message that is to be forwarded to all other members of theprocess group. Unlike the group state value, this message is notpersistent. Once it is shown to the group members, Group Services nolonger keeps track of it. However, Group Services does guaranteedelivery to all non-failed group providers.

The message can be used by a group provider, for instance, to forwardsignificant information during the protocol that cannot be carried bythe other responses within a vote. For example, it can be used toprovide information that cannot be reflected in the provider's votevalue or to provide information that does not need to be madepersistent. In one example, it can inform the group members of aparticular function to perform.

In accordance with one embodiment of the present invention, eachprovider of a process group is expected to vote at a voting phase of aprotocol. Until all of the providers vote, the protocol remainsuncompleted. Thus, a mechanism is provided in the voting protocol, inaccordance with the principles of the present invention, in order tohandle the situation in which one or more providers have not provided avote. In particular, the voting mechanism includes a default vote value,which is explained in detail below.

As examples, a default vote value is used when a provider fails duringthe execution of the protocol or when the processor in which theprovider is executing fails or if the provider becomes non-responsive,as described herein. The default vote value guarantees forward progressfor the protocol and for the process group. A process group initializesits default vote value when the group is first formed by, for example,its attributes. In one embodiment, the default vote value can either beAPPROVE or REJECT. During each voting phase, the default vote value canbe changed to reflect changing conditions within the group.

In the situation in which a process fails during the protocol, GroupServices determines this, as described above, and thus, at any votingphase for the protocol, the group leader will submit the group's currentdefault vote for the failed process. Similarly, if Group Servicesdetermines that the processor executing a member provider has failed,then the group leader once again submits a default vote.

If, however, a processor or process is available but non-responsive,then the default vote value can also be used. In one example, a processis deemed non-responsive when it does not respond to a vote within atime limit set by the process group for that protocol. (Each protocolfor each process group can have its own time limit.) When the process isnon-responsive, the default vote value assigned to the process group isused by the group leader for this particular process. In one embodiment,it is possible to have no time limit. In that situation, Group Serviceswill wait until the provider eventually responds or until it fails.

In one embodiment, when a default vote is used, the providers areinformed of this.

In accordance with the principles of the present invention, a providercan dynamically update the default vote value at any one or more of thevoting steps within the protocol. This allows flexibility in thehandling of failures, as the protocol progresses. The proposed defaultvalue is submitted along with the vote value of the process. The newdefault vote value remains in effect for the remainder of the protocol,unless another default vote value is proposed at a later voting step. Ifmultiple default vote values are proposed at a particular voting step,then in one embodiment, Group Services (i.e., the group leader) selectsthe value submitted by the first process to respond. Once the protocolis complete, the default vote value for the process group reverts backto the value initially set for the group.

A default vote value is treated in the same manner as any other votevalue. However, default vote values cannot, in one embodiment, includeother information for the vote, such as, for instance, a message, agroup state value or a new proposed updated default vote value.

As described above with reference to FIG. 11, all of the above-describedproposed protocols can be proposed as one-phase protocols in which theprotocol is proposed and accepted in one multicast. Therefore, it is notnecessary to take a vote.

Described in detail above are mechanisms for ensuring highly-availablemulticomputer applications. As one example, the mechanisms of thepresent invention can be used for providing a fault-tolerant andhighly-available system. The mechanisms of the present inventionadvantageously provide a general purpose facility for coordinating,managing and monitoring changes to the state of process groups executingwithin the system.

In accordance with the principles of the present invention, membershipwithin processor groups and process groups can be dynamically updated.In both cases, processors or processes can request to be added orremoved from a group. The mechanisms of the present invention ensurethat these changes are performed consistently and reliably.

Additionally, in accordance with the principles of the presentinvention, mechanisms are provided for enabling messages to be sent toone or more particular groups of processors, without having to send themessages to all of the processor groups. Each processor group has theability to monitor and manage its own set of messages and fordetermining if one or more messages has been missed. If a message hasbeen missed, that message is then retrieved from another member of thegroup. There is no need to maintain stable storage for these messages.Each member of the group has the messages, and thus, can provide missingmessages to other members. This advantageously reduces the costs ofhardware.

Further, in accordance with the principles of the present invention,mechanisms are provided for recovering from a failed group leader. Thesemechanisms ensure that a new group leader is selected easily andefficiently.

The mechanisms of the present invention also provide an applicationprogramming interface that unifies a number of protocols into onesingle, integrated framework for the processes. As one example, theintegrated application programming interface provides a facility forcommunicating between members of process groups, as well as a facilityfor synchronizing processes of a process group. Additionally, the sameinterface provides a facility for dealing with membership changes toprocess groups, as well as changes to group state values.

The application programming interface also includes a mechanism thatenables Group Services to monitor the responsiveness of the processes.This can be performed in a similar fashion as to a ping mechanism usedin computer network communications.

In addition to the above, the mechanisms of the present inventionprovide a dynamic barrier synchronization technique. In accordance withthe principles of the present invention, the number of synchronizationphases included in any one protocol is variable, and can be determinedby the members voting on the protocol.

The mechanisms of the present invention can be included in one or morecomputer program products including computer useable media, in which themedia include computer readable program code means for providing andfacilitating the mechanisms of the present invention. The products canbe included as part of a computer system or sold separately.

The flow diagrams depicted herein are just exemplary. There may be manyvariations to these diagrams or the steps described therein withoutdeparting from the spirit of the invention. For instance, the steps maybe performed in a differing order, or steps may be added, deleted ormodified. All of these variations are considered a part of the claimedinvention.

Although preferred embodiments have been depicted and described indetail herein, it will be apparent to those skilled in the relevant artthat various modifications, additions, substitutions and the like can bemade without departing from the spirit of the invention and these aretherefore considered to be within the scope of the invention as definedin the following claims.

What is claimed is:
 1. A communications method for a distributedcomputing environment comprising:sending a first message to a firstgroup of processors and a second message to a second group ofprocessors; maintaining, by said first group of processors, said firstmessage and, by said second group of processors, said second message;said sending said first message comprising; sending a request to a firstgroup leader of said first group of processors to multicast said firstmessage; assigning, by said first group leader, a sequence number tosaid first message providing a first ordered message; and multicasting,by said first group leader, said first ordered message to said firstgroup of processors; recovering from a failure of said first groupleader including determining a new group leader; and wherein saiddetermining comprises obtaining from a membership list ordered insequence of joins of processors to said first group of processors saidnew group leader, said new group leader being a next processor in saidmembership list.
 2. The communications method of claim 1, furthercomprising caching, by said first group of processors, said firstmessage until a predefined condition occurs.
 3. The communicationsmethod of claim 2, wherein said predefined condition comprises receivingan acknowledgment that said first message has been received by each ofsaid processors of said first group of processors.
 4. The communicationsmethod of claim 1, wherein said first group of processors is executing afirst group of one or more related processes and said second group ofprocessors is executing a second group of one or more related processes.5. The communications method of claim 1, wherein said sending comprisessending a first plurality of messages to said first group of processorsand a second plurality of messages to said second group of processors,and wherein said maintaining comprises maintaining a first sequenceorder, by said first group of processors, for said first plurality ofmessages and a second sequence order, by said second group ofprocessors, for said second plurality of messages, said first sequenceorder and said second sequence order being independent of one another.6. The communications method of claim 1, further comprising:determining,by each processor of said first group of processors, whether said firstordered message received via said multicasting is in proper sequenceorder, wherein an improper sequence order indicates a missing message;and requesting, from a processor of said first group of processors, saidmissing message when said sequence order is improper.
 7. Thecommunications method of claim 1, further comprising determining saidfirst group leader, said determining comprising sending a request to aname server requesting identification of said first group leader.
 8. Thecommunications method of claim 1, wherein said determining comprisesrequesting appointment of said new group leader from a name server, saidname server obtaining said new group leader from said membership list.9. The communications method of claim 8, wherein said name server is aprocessor within said first group of processors.
 10. The communicationsmethod of claim 8, wherein said name server is a processor independentof said first group of processors.
 11. The communications method ofclaim 8, wherein said membership list is located within said nameserver.
 12. The communications method of claim 1, wherein saidmembership list is included in each processor of said first group ofprocessors.
 13. The communications method of claim 1, wherein said firstgroup of processors and said second group of processors have one or moreprocessors in common.
 14. The communications method of claim 1, whereinsaid sending said first message comprises:initiating said first messageby a group leader of said first group of processors; assigning, by saidgroup leader, a sequence number to said first message providing a firstordered message; and multicasting, by said group leader, said firstordered message to said first group of processors.
 15. Thecommunications method of claim 14, further comprising:determining, byeach processor of said first group of processors, whether said firstordered message received via said multicasting is in proper sequenceorder, wherein an improper sequence order indicates a missing message;and requesting, from a processor of said first group of processors, saidmissing message when said sequence order is improper.